Estimating the Accuracy of Learned Concepts

This paper investigates alternative estimators of the accuracy of concepts learned from exam­ ples. In particular, the cross-validation and 632 bootstrap estimators are studied, using syn­ thetic training data and the FOIL learning al­ gorithm. Our experimental results contradict previous papers in statistics, which advocate the 632 bootstrap method as superior to crossvalidation. Nevertheless, our results also sug­ gest that conclusions based on cross-validation in previous machine learning papers are unreli­ able. Specifically, our observations are that (i) the true error of the concept learned by FOIL from independently drawn sets of examples of the same concept varies widely, (i i) the esti­ mate of true error provided by cross-validation has high variability but is approximately unbi­ ased, and (i i i) the 632 bootstrap estimator has lower variability than cross-validation, but is systematically biased.